Assessing the Generalizability of Deep Learning Models Trained on Standardized and Nonstandardized Images and Their Performance Against Teledermatologists: Retrospective Comparative Study

Background Convolutional neural networks (CNNs) are a type of artificial intelligence that shows promise as diagnostic aid for skin cancer. However, the majority trained using retrospective image data sets with varying capture standardization. Objective The aim our study was to use CNN models same architecture—trained on acquired either device and technique (standardized) or varied devices techniques (nonstandardized)—and test variability in performance when classifying cancer images different populations. Methods In all, 3 CNNs architecture were trained. nonstandardized (CNN-NS) 25,331 taken from International Skin Imaging Collaboration (ISIC) devices. standardized (CNN-S) 177,475 MoleMap device, number 2 (CNN-S2) subset (matched classes training CNN-NS). These then tested external sets: 569 Danish images, publicly available ISIC 2020 set consisting 33,126 University Queensland (UQ) 422 images. Primary outcome measures sensitivity, specificity, area under receiver operating characteristic curve (AUROC). Teledermatology assessments used determine model compared teledermatologists. Results When CNN-S achieved an AUROC 0.861 (95% CI 0.830-0.889) CNN-S2 0.831 0.798-0.861; models), both outperforming CNN-NS (nonstandardized model; P=.001 P=.009, respectively), which 0.759 0.722-0.794). additional (ISIC UQ), (P<.001 P<.001, respectively) (P=.08 P=.35, still outperformed CNN-NS. matched mean sensitivity specificity teledermatologists set, models’ resultant sensitivities specificities surpassed by CNN-S, differences not statistically significant (sensitivity: P=.10; specificity: P=.053). Performance across all well influenced quality. Conclusions had improved and, therefore, greater generalizability classification applied unseen sets. This finding is important consideration future algorithm development, regulation, approval.